Robust Visual Servoing under Human Supervision for Assembly Tasks

Abstract

We propose a framework enabling mobile manipulators to reliably complete pick-and-place tasks for assembling structures from construction blocks. The picking uses an eye-in-hand visual servoing controller for object tracking with Control Barrier Functions (CBFs) to ensure fiducial markers in the blocks remain visible. An additional robot with an eye-to-hand setup ensures precise placement, critical for structural stability. We integrate human-in-the-loop capabilities for flexibility and fault correction and analyze robustness to camera pose errors, proposing adapted barrier functions to handle them. Lastly, experiments validate the framework on 6-DoF mobile arms.

Media

Picture of the algorithmic setup for picking and placing.

BibTex

@misc{ECC25_visualServo,
      title={Robust Visual Servoing under Human Supervision for Assembly Tasks}, 
      author={Victor Nan Fernandez-Ayala and Jorge Silva and Meng Guo and Dimos V. Dimarogonas},
      year={2025},
      eprint={2504.12506},
      archivePrefix={arXiv},
      primaryClass={eess.SY},
      url={https://arxiv.org/abs/2504.12506},
}